Pictorial digital image processing incorporating adjustments to compensate for dynamic range differences

ABSTRACT

Image-dependent tone and color reproduction processing of digital images is accomplished by creating a spatially blurred and sub-sampled version of the original image, and applying knowledge of the capture device physical characteristics to obtain statistics related to the scene or original captured. The statistics are used for image-dependent linearization of the captured image data based on an OECF model, and in conjunction with information about the output medium, to calculate image-specific preferred tone reproduction curves based on a preferred reproduction model. The image data can then be processed for reproduction on output media with different density ranges and color gamuts. All the processing described can be accomplished automatically, but the access to accurate scene information afforded by the image-dependent linearization, and to the perceptually intuitive parameters controlling the calculation of the preferred reproduction curves, also allows for simple, intuitive manual adjustment.

RELATED APPLICATIONS

This application is a Division of allowed parent application Ser. No.09/771,735, filed Jan. 29, 2001, by Jack M. Holm, entitled PICTORIALDIGITAL IMAGE PROCESSING INCORPORATING IMAGE AND OUTPUT DEVICEMODIFICATIONS, which parent application is a Division of applicationSer. No. 08/822,053, filed Mar. 24, 1997, by Jack M. Holm, entitledSTRATEGY FOR PICTORIAL DIGITAL IMAGE PROCESSING, now U.S. Pat. No.6,249,315. The parent and grandparent applications are herebyincorporated by reference.

TECHNICAL FIELD

This invention relates generally to the processing of digital images toproduce desired tone and color reproduction characteristics.Specifically, this invention makes use of capture and output deviceinformation, in conjunction with opto-electronic conversion function(OECF) and preferred reproduction models, to determine image-specificprocessing based on statistics derived from the image data. Theprocessing determined may be applied automatically or with user input.

BACKGROUND OF THE INVENTION

Digital cameras and scanners are used to capture image data from a largevariety of scenes and originals. A number of automatic approaches areemployed to process this data for reproduction; but when reproductionquality is critical, most images are processed manually by experts.Expertly processed images are also overwhelmingly preferred, even byinexperienced viewers, when comparisons are made.

Manual processing is time consuming and must be done by individuals withsignificant expertise, partly because the controls found in currentlyavailable software packages make it difficult to achieve desired toneand color reproduction. Simple controls tend to vary the reproduction inways that miss the optimum, and complex controls offer too many degreesof freedom. If a way could be found to produce results similar to thoseproduced by experts, either automatically or with simple and intuitivemanual adjustments, digital photography would become a much moreattractive alternative to conventional photography.

The practice of conventional photography suggests that improvements inthis direction are possible. Currently, conventional photographs tend tobe superior to automatically processed digital photographs in tone andcolor reproduction, with photographs processed at professionallaboratories being far superior. Yet the flexibility of digital systemsis greater than that of conventional systems. Digital photographs havethe potential to be better than conventional photographs, and expertlyprocessed digital photographs are currently at least as good. Digitalprocessing approaches that mimic the relatively fixed behavior ofconventional photographic systems should be straightforward to develop.Insight into digital processing approaches can also be obtained byexamining what experts do manually. If this is done, it is found thatmost of the decisions made are based on evaluations of the image withrespect to the scene or original and with the desired reproduction goalin mind. It should be possible to develop software algorithms that canperform these evaluations and process images accordingly.

Three major factors have hindered progress in this area. The first isthat expert manual processing is almost always image-dependent and isbased on understood tone and color reproduction objectives; but thedevelopment of most digital tone and color reproduction processing hasfocused on schemes which do not consider the image data, or consider itwithout regard for established pictorial considerations. The second isthat the exact meaning of the image data, with respect to the scene ororiginal, must be known. To date, the approaches used have ignored manynon-linearities, such as those introduced by optical flare and otherimage capture effects, and have concentrated on techniques based almostexclusively on colorimetry. Colorimetry is strictly applicable only whenthe capture spectral sensitivities are color matching functions, or whenthe colorants used in the original are known and limited to a numberwhich is not greater than the number of spectral capture channels. Withdigital cameras in particular, this is frequently not the case. Otherdifficulties in determining scene physical characteristics have resultedfrom a lack of standard, accurate measurement approaches. When basicflaws are present in a measurement approach, such as the omission offlare considerations and the fact that the spectral characteristics ofthe detector preclude calorimetric information from being obtained,attempts to calculate scene values inevitably produce erroneous results.These errors reduce accuracy expectations and mask other error sources,seriously degrading the correlation between captured data and scenecharacteristics.

The final factors which have hindered progress are the slow recognitionof the need for preferred reproduction as an alternative goal tofacsimile reproduction, that preferred reproduction is dependent on thescene or original, and that the output and viewing conditioncharacteristics. As mentioned previously, most digital tone and colorreproduction processing development has focused on schemes which do notconsider the image data. Also, color management approaches based oncolorimetry attempt to produce reproductions with colorimetric valuessimilar to those of the original. While colorimetric measures canconsider some viewing condition effects, others are not considered andthe effects of the scene characteristics and media type on preferredreproduction are ignored.

It is clear that the factors limiting tone and color reproductionquality in digital images stem from an incomplete and sometimesinappropriate global strategy. The inventor has attempted to deal withthese problems in two ways: through participation and leadership in thedevelopment of national and international standards and with theinventions presented here. The following list specifies gaps in thestrategy and the attempts to fill them in:

1. Inaccurate and non-standard device measurements.

Image capture and output devices are measured in a variety of ways, withvarious measurement and device effects being ignored. Specifically:

a) Flare and other non-linearities in both capture devices and measuringinstruments are frequently not considered, or are measured for aparticular condition, and the resulting values are erroneously assumedto be applicable to other conditions.

b) Test targets captured typically have considerably lower luminanceratios than pictorial scenes, so the extremes of the capture devicerange are truncated or left uncharacterized.

c) Attempts are made to correlate image data to calorimetric quantitiesin scenes by capturing data of test targets with devices whose channelspectral sensitivities are not color matching functions. Correlationsestablished in this fashion will depend on the test target used and maynot apply to image data from other subjects.

d) Measurement precision is frequently specified in linear space,resulting in perceptually large errors for darker image areas. Theseareas are also most affected by flare, making dark area measurementsparticularly inaccurate.

e) Measurement geometries and lighting may be inconsistent orinappropriate to the device use conditions.

All of these effects compound to produce device characterizationmeasurements which can be quite inaccurate. A common perception of theseinaccuracies is that it is practically impossible to obtain stablemeasurements and, therefore, measurement accuracy need not be too high.Frequently, assumptions are made about the nature of image data, becauseit is felt that the assumption will be as close to the real value as ameasurement.

However, in conventional photography, standardized density measurementtechniques have evolved over decades. These techniques routinely producerepeatable measurements with several orders of magnitude higher accuracythan those obtained for digital systems, which is one of the reasons theless flexible conventional systems are able to outperform currentautomatic digital systems. Unfortunately, the reason these techniquesare so accurate is because they have been refined specifically forconventional photographic materials. A great deal of work will berequired to develop similar techniques for the devices and material usedin digital systems.

Work has just begun in this area, but significant progress is alreadybeing made. A new standard to be issued by the InternationalOrganization for Standardization (ISO), “ISO 14524,Photography—Electronic still picture cameras—Methods for measuringopto—electronic conversion functions (OECFs),” is almost complete, andwork has begun to develop two new standards: “Digital still picturecameras—Methods for the transformation of sensor data into standardcolour spaces” and “Photography—Film and print scanners—Methods formeasuring the OECF, SFR, and NPS characteristics.” While these effortsare collaborative and open, the inventor is the project leader for theformer two efforts, and participates in the latter.

2. Difficulties in communicating device information due to a lack ofstandard data types, terms, and data formats.

Even if accurate measurements are available, a complete processingstrategy requires that the measurements characterize the device inquestion by completely filling with values an enumerated list ofexpected measurements. These values must also be provided in the imagefile in a standard format to be readily usable by a variety ofprocessing software. “ISO 12234/1, Photography—Electronic still picturecameras—Removable memory, Part 1: Basic removable memory referencemodel” and “ISO 12234/2, Photography—Electronic still picturecameras—Removable memory, Part 2: Image Data Format—TIFF/EP” define thecharacterization data required and where and how it is to be included inimage files. The inventor was a key participant in the development ofthis standard, particularly the enumeration of the characterizationdata. It should be noted that because of a lack of consensus concerningthe need for some of the data among the members of the ISO workinggroup, some of the data categories are defined as “optional” at present,but the necessary categories are described.

3. How to deal with image-dependent capture non-linearities.

The measurement methods described in the standards mentioned above tellhow to measure digital photography system characteristics using varioustest targets, but they do not deal with methods for estimatingimage-dependent capture non-linearities. A solution to this problem isdescribed in this patent application.

4. Lack of specification of standard, optimal methods for transformingimage data from the capture device spectral space to standard colorspaces.

A number of methodologies have evolved for transforming capture devicedata into intermediate or standard color spaces. Many of these methodshave merit in particular circumstances, but in many cases are appliedinappropriately. The lack of accurate characterization data compoundsthe problem in that it is difficult to tell if the cause of low qualitytransformed data is an inappropriate transformation method, inaccuratecharacterization data, or both.

Another difficulty has been that, until recently, the only standardcolor spaces used for digital photography were those defined by the CIE(Commission Internationale de L'Éclairage or International Commission onIllumination) based on the human visual system (HVS). For variousreasons, it is generally impractical to design digital photographysystems that mimic the HVS. Most digital photography systems analyzered, green, and blue (RGB) light; and most output devices modulate thesespectral bands. In conventional photography, specific RGB bands are welldefined by the spectral characteristics of the sensitizing dyes andcolorant dyes used and by standards such as “ISO 7589,Photography—Illuminants for Sensitometry—Specifications for Daylight andIncandescent Tungsten” (which defines typical film spectralsensitivities) and “ISO 5/3, Photography—Density measurements—Spectralconditions.” These standards were developed many years ago, but theinventor actively participates in their maintenance and revision.

In digital photography, a wide variety of spectral sensitivities andcolorants are used by different systems. Many of these systems are basedon RGB analysis and synthesis, but the data produced in capturing aparticular scene can vary substantially between capture devices. Also,if the same RGB image data is provided to different output devices,significantly different results will be obtained; and the differenceswill not be, for the most part, the results of optimization of the imagedata to the media characteristics.

Over the past five years, companies involved with digital imaging haverecognized this problem and invested significant resources in solvingit. Some progress has been made, particularly within the InternationalColor Consortium (ICC), an association comprising most of the majorcomputer and imaging manufacturers. However, the efforts of the ICC havebeen directed at producing consistent output from image data. Themetrics employed are based on colorimetry and generally aim to produceoutput on different devices that is perceptually identical when viewedunder a standard viewing condition. This aim is commonly referred to as“device-independent color.” Device-independent color is an appropriategoal in some cases, but frequently falls short. Different media havevastly different density range and color gamut capabilities, and theonly way to make sure that all colors are rendered identically on allmedia is to limit the colors used to those of the lowest dynamic range(density range and color gamut) medium. This is certainly not desirable,and consequently a number of ICC member (and other) companies are nowcreating “ICC profiles” that produce colors from the same image datawhich vary between devices. (ICC profiles are device-specifictransformations in a standard form that ostensibly attempt to transformimage data to produce device-independent results.)

The basis for the color science on which device-independent color isbased is the behavior of the HVS. Much of this behavior is reasonablywell understood, but some is not, particularly the effects of some typesof changes in viewing conditions, and localized adaptation. Also, theeffects of media dynamic range on preferred reproduction have little todo with the HVS. Appearance models may be able to predict how somethingwill look under certain conditions, but they give no information aboutobserver preferences for tone and color reproduction in a particularphotograph.

ICC profiles are currently being produced that attempt to transformcaptured image data to produce calorimetric values (input profiles), andthat take image data and the associated input profile and attempt totransform the calorimetric values to data suitable for output on aparticular device (output profiles). These profiles are generallyconsidered to be device-specific, in that a particular input profile isassociated with a particular capture device and a particular outputprofile is associated with a particular output device. This type ofassociation makes sense in view of the philosophy of the approach. If ascene or original is characterized by particular calorimetric values,the goal of the input profile is to obtain these values from thecaptured data. Likewise, the goal of the output profile is to reproducethe colorimetric values on the output medium. Since the goal isfacsimile calorimetric reproduction, the profile should be independentof the scene content or media characteristics.

If the capture device spectral sensitivities and/or colorants used inthe original make it possible to determine calorimetric values fromcaptured data, it is theoretically possible for ICC-type input profilesto specify the appropriate transformations. Also, if the characteristicsof the capture device do not vary with the scene or original capturedand the device spectral sensitivities are color matching functions, asingle profile will characterize the device for all scenes or originals.If knowledge of the colorants is required to allow calorimetric data tobe obtained, a single profile is adequate for each set of colorants.Unfortunately, flare is present in all capture devices that form animage of the scene or original with a lens (as opposed to contact typeinput devices, like drum scanners). The amount of flare captured willvary depending on the characteristics of the scene or original.Occasionally, other image-dependent non-linearities are alsosignificant. For ICC profiles to specify accurate transformations fordevices where flare is significant, not only must the colorimetricspectral conditions be met, but the image-dependent variability must bemodeled and considered. The resulting input profiles are dependent onthe distribution of radiances in the scene or original, as well as thecapture device used and the colorants (if applicable).

In summary, the primary difficulties with using ICC profiles to specifytransformations are:

a) ICC input profiles only allow transformations to CIE color spaces,yet transformations to this type of color space are valid only if thecapture device sensitivities are color matching functions, or colorantsfound in the scene or original are known, and are spanned by spectralbasis functions not greater in number than the number of device spectralcapture channels. These conditions are almost never met when digitalcameras are used to capture natural scenes.

b) The appropriate ICC input profile for a particular device and/or setof colorants to be captured is generally assumed to be invariant withthe content of the scene or original. This assumption is not valid withthe many capture devices that have significant amounts of flare, such asdigital cameras and area array scanners.

c) The measurement techniques used to determine ICC profiles arevariable, and the profiles constructed are usually not optimal.Frequently, profiles are not linearized correctly, neutrals are notpreserved, and incorrect assumptions are made about the colorants in thescene or original. These inaccuracies are masked by and compounded withthe inaccuracies mentioned in a and b.

d) While it is recognized that different output media must producedifferent calorimetric renderings of the same image data for the resultsto be acceptable, there is no standard methodology for determining howto render images based on the dynamic range of the scene or original, ascompared to the output medium.

The ICC efforts have resulted in a significant improvement over doingnothing to manage colors, but in their current manifestation are notviewed as a substitute for manual processing. Fortunately, the ICCapproach is continuing to evolve, and other organizations are alsocontributing. In particular, there is a proposal in the ICC to allowanother standard color space based on a standard monitor. This colorspace is an RGB space, making it more appropriate for use with manycapture devices, particularly RGB-type digital cameras and filmscanners. This proposal is also being developed into a standard:“CGATS/ANSI IT8.7/4, Graphic technology—Three Component Color DataDefinitions.” The inventor is involved in the development of thisstandard. Also, the proposed new ISO work item mentioned previously, forwhich the inventor is the project leader, “Digital still picturecameras—Methods for the transformation of sensor data into standardcolour spaces,” is specifically aimed at specifying methods fordetermining optimal transformations. As these efforts are completed andif the methods for dealing with image-dependent non-linearities of myinvention are used, it should become possible to specify capture devicetransformations that are determined in standard ways and based onaccurate measurements.

5. How to determine preferred reproduction based on the characteristicsof the scene or original and the intended output medium.

The first part of the digital image processing pipeline transformscapture device data into a standard color space. Once the data is insuch a color space, it is necessary to determine an outputtransformation that will produce preferred reproduction on the outputmedium of choice. A method for accomplishing this is contemplated by myinvention.

6. How to take image data processed for preferred reproduction on oneoutput medium and transform it for preferred reproduction on anotheroutput medium.

Frequently, it is necessary to take image data which has already beenprocessed for preferred reproduction on one output device and process itfor preferred reproduction on another output device. A method foraccomplishing this also contemplated by my invention.

7. How to implement user adjustments that produce preferred reproductionwith maximum simplicity and intuitiveness.

As stated previously, current manual processing software tends to beoverly complicated and offers too many degrees of freedom or isincapable of producing optimal results. The implementation of thepreferred reproduction model described in this patent application allowsfor user specification of key parameters. These parameters are limitedin number; and changing them produces transformations which alwaysproduce another preferred rendering, limiting the possible outputs tothose that are likely to be preferred.

SUMMARY OF THE INVENTION

Embodiments of my invention, in conjunction with the above-mentionedinternational standards under development, solve the above-identifiedproblems by providing a complete strategy for the processing of digitalimage data to produce desired tone and color reproductioncharacteristics. The details of this strategy are as follows:

1. A scaled version of the image is constructed by spatially blurringand sub-sampling each channel of the image data. The scaled version ispreferably a reduced version, but can be of any scale with respect tothe original image. The blurring and sub-sampling are accomplished usingone or more filters that first blur the image data using a blur filterwith a radius that is primarily related to the number of pixels, rows ofpixels, or columns of pixels in the image channel, but can also beaffected by other factors, such as the intended output size or pixelpitch, the intended output medium, the numerical range of the imagedata, etc. Any common blur filter can be used, such as a boxcaraveraging or median, a Gaussian blur, etc. The blurred image is thendecimated to produce the scaled image, which is stored for future use.

2. The capture device focal plane OECFs are determined for each channelaccording to ISO 14524 for digital cameras or the standard which resultsfrom the new work item under development for scanners. The inverses ofthese OECFs are then determined, either in functional form or aslook-up-tables (LUTs). This information may also be provided by thedevice manufacturer or included in the image file header with some fileformats.

3. The scaled image data is transformed into focal plane data using theinverse focal plane OECFs. Statistical values are then determined foreach channel from the transformed data. Typical statistical values arethe minimum and maximum focal plane exposures, the mean focal planeexposure, and the geometric mean focal plane exposure. Other statisticalvalues may be determined in some cases.

4. The capture device design and OECFs are evaluated to determine if thecapture device has significant image-dependent non-linearities or flare.If image-dependent effects are found, they are modeled. The model to beproduced should predict the amounts of non-linearities and flare basedon statistical values determined from the scaled image data. Models canbe constructed by capturing a variety of scenes or originals (such asISO camera OECF charts with a variety of luminance ranges and backgroundluminances), determining the flare and non-linearities encountered whencapturing these charts, and then correlating the measured values withthe scaled image statistics. Flare models can also be constructed bycompounding extended point-spread-functions. A flare model may beprovided by the device manufacturer, but there is no mechanism atpresent for including this information in the file format.

5. The estimated camera or scanner OECFs for the image represented bythe scaled image are determined for each channel using the OECFmeasurement standards mentioned, in conjunction with the flare andnon-linearity model. The inverses of these OECFs are then determined,either in functional form or as LUTs. These inverse OECFs, which will bereferred to as the input linearization information or inputlinearization tables, are stored for future use.

6. The capture device spectral sensitivities are evaluated and anappropriate transformation to an intermediate spectral or color space isdetermined. This intermediate color space is preferably a color spaceappropriate for application of the preferred reproduction model, such asa standard color space. If the intermediate color space is a standardcolor space, the transformation can be determined according to one ofthe methods in the proposed new ISO standard. In this case, the inputlinearization table is used to linearize the captured data, as requiredby the standard. The transformation may also be provided by the devicemanufacturer or included in the image file header with some fileformats.

7. The scaled image is linearized using the input linearization tableand transformed to the intermediate color space using the transformationdetermined. A luminance channel image is then determined using theequation appropriate for the intermediate color space. Statisticalvalues are then determined from the luminance channel data. Typicalstatistical values are the minimum and maximum (extrema) sceneluminances, the mean luminance, and the geometric mean luminance. Otherstatistical values may be determined in some cases. The scaled imagedata is generally not needed after these statistical values aredetermined.

8. The output device is determined, either by assuming it to be astandard monitor, by asking the user, or by the software (if intendedfor a specific device). The visual density ranges of all selectableoutput devices should be known. The viewing conditions under which theoutput will be viewed may also be specified.

9. The statistical values determined from the luminance channel of thescaled image, the density range of the output device, and the viewingillumination level (if known) are input to the preferred reproductionmodel. This model calculates an image and output specific preferred tonereproduction curve. This tone reproduction curve is typically applied toRGB channels, to produce preferred tone and color reproduction.

10. The output device electro-optical conversion function (EOCF)characteristics are determined by measuring the output of the device forall possible input digital levels or, in the case of the standardmonitor, by using the standard monitor EOCF. An output transformation isthen determined by combining the preferred tone reproduction curve withthe output device EOCF. This transformation may be expressed infunctional form or as a LUT and will be referred to as the output table.

11. The image data for the entire image is linearized using the inputlinearization tables. It is then preferably transformed into theintermediate color space. This color space can be a standard RGB space,although monochrome image data should be transformed into aluminance-type space, and this processing also may be used to producedesired tone reproduction characteristics with luminance-chrominancetype color space data. The output tables are then applied to the linearintermediate color space data to produce digital code values appropriatefor a standard monitor or the specified output device. If necessary,standard RGB values or other color space values corresponding topreferred reproduction may be converted to another color space for useby the output device. In this case, the goal of the processing employedby the output device is to produce a facsimile reproduction of thepreferred reproduction as expressed in the standard RGB or otheroriginal color space. The preferred reproduction should have beendetermined with consideration of the luminance range capabilities of theoutput medium.

DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic flow chart of the overall method of the invention.

FIG. 2 is a schematic flow chart of the step of gathering preliminaryinformation illustrated in FIG. 1.

FIG. 3 is a schematic flow chart of the step of providing an originalimage illustrated in FIG. 1.

FIG. 4 is a schematic flow chart of the step of constructing a scaledimage from or a scaled version of the original image illustrated in FIG.1.

FIG. 5 is a schematic flow chart of the step of analyzing the scaledimage or scaled version illustrated in FIG. 1.

FIG. 6 is a schematic flow chart of the step of processing the originalimage to produce a processed or output image illustrated in FIG. 1.

FIG. 7 is a schematic flow chart of another embodiment of the invention.

FIG. 8 is a schematic flow chart of another embodiment of the invention.

FIG. 9 is a schematic flow chart of another embodiment of the invention.

FIG. 10 is a schematic flow chart expanding on the embodiment shown inFIG. 9.

FIG. 11 is a schematic flow chart expanding on the embodiment shown inFIG. 9.

FIG. 12 is a schematic flow chart of another embodiment of theinvention.

FIG. 13 is a schematic flow chart expanding on the embodiment shown inFIG. 12.

FIG. 14 is a schematic flow chart expanding on the embodiment shown inFIG. 12.

FIG. 15 is a schematic flow chart of another embodiment of theinvention.

FIG. 16 is a schematic flow chart of another embodiment of theinvention.

FIG. 17 is a schematic flow chart of another embodiment of theinvention.

DESCRIPTION OF THE INVENTION

The processing strategy described herein and shown schematically in theaccompanying Figures and Tables seeks to accomplish four goalssimultaneously:

Process image data to produce the best possible result in terms of whatis desired by the user.

Minimize complexity whenever possible in order to reduce computationalrequirements and emphasize the basic function of the processingalgorithms employed.

Automate the processing to the greatest extent that is consistent withhardware capabilities and user quality expectations.

Improve the efficiency of user adjustments by focusing capabilities onthe more likely outcomes and making the adjustment process as intuitiveas possible.

The above goals force processing strategies in specific directions. Inparticular, it is desirable to consider the physics of imaging systems.Many operations are best performed with the image data in a particularphysical representation. Also, physical measurements of the behavior ofcomponents in each system can be extremely useful in determiningprocessing parameters. Since little of this information is obtained bythe user, it is desirable to automate the transfer of this information,either as part of the image file or between devices and the processingsoftware. Several newer image file formats accommodate this transfer.

Another consideration in the development of the processing strategy isdevice-independent performance optimization. Digital image data comesfrom a variety of sources and may be used for a variety of purposes. Forany strategy to be truly useful, it must be able to produce excellentresults on a large variety of devices. Device-independent performanceoptimization, however, should not be confused with most currentmanifestations of device-independent color. Optimized performanceoccasionally results from reproducing colorimetric measurements; butfrequently an optimized reproduction will be somewhat different from theoriginal, particularly with photographs. Some of these differences areattributable to differences in human visual system adaptation.Development of a truly comprehensive appearance model and thereproduction of appearance would undoubtedly produce optimizedreproductions in many cases. Such a model would by necessity beextremely complex, however, and might require data about surround andviewing conditions, etc., that are difficult to obtain. Also, in somecases, even the reproduction of appearance might not produce thepreferred result.

For many decades, photography has evolved an empirical set of preferredreproduction goals. These goals have been driven to some extent bymaterials considerations, but the relatively high quality ceiling of thephotographic process prevents media limitations from greatly affectingthe goals. A more significant problem is that the goals were notextensively documented. Also, the relative rigidity of chemicalprocesses prevents the goals from being tweaked in ways that would beadvantageous with more flexible systems, such as digital systems.Nevertheless, the implementation of these goals in conventionalphotographic systems has resulted in pictorial imaging systems capableof producing preferred reproductions of excellent quality and relativeinsensitivity to changes in the viewing environment.

Recently, attempts have been made by the inventor to document preferredphotographic reproduction goals and extend them for application todigital processing. A result of this work is the emergence of severalissues commonly considered to be of major importance in photography. Inparticular, the effects of flare, image key (high- or low-), scenedynamic range, viewing conditions, and veiling glare are addressed.Addressing these issues for device-independent performance optimizationin digital photography requires that the proposed processing strategy bescene and output viewing condition dependent. Photography deals withoutput viewing conditions through the use of specific media and standardrecommendations for specific applications. Scene dependent issues aredealt with by engineering materials for graceful failure and by humanintervention at both the image capture and processing stages. Withdigital systems, it is possible-to shift the scene dependentintervention to smart processing algorithms.

Tone and color processing are of major importance in producing excellentimages, but spatial processing can also have a significant effect onquality. The expense of manufacturing one-shot digital cameras withadequate numbers of pixels and the use of color filter arrays hasincreased the importance of spatial issues even further. Over the pastdecade, research in spatial processing has been largely separate fromcolor processing, but device-independent performance optimizationrequires that the two be integrated. Optimized spatial processing isalso becoming more output dependent. The spatial frequency capabilitiesof the output media and the viewing distance affect the degree ofsharpening desired. Photographic artists and engineers have long knownthat mean-square-error (Wiener filter) based restoration is a starttoward optimized processing, but that additional edge enhancement isneeded. Recent work in information throughput based restoration isbeginning to suggest new mathematical approaches to spatial processing.

In discussing processing strategies, it is important to differentiatebetween pictorial processing to produce specific reproduction goals,image editing, and image manipulation. Image editing implies directmanual alteration of the image data to produce some desired result. Withimage manipulation, all or part of an image is intentionally altered toproduce a specific effect or make some point. Moving objects around,changing peoples faces, radically changing the colors of objects, anddistortions are image manipulation. Taking an image and processing it toproduce a pleasing result is pictorial processing. The boundary betweenthe two can blur when considering how much of an increase in contrast orsaturation is preferred, as opposed to exaggerated. Most popularphotographic image processing applications are well suited for imageediting and manipulation. The processing strategy outlined here isoriented toward pictorial processing.

Reproduction Goal Choices

The first step in the processing of pictorial images is to choose thedesired reproduction goal. The goal chosen needs to be realizable withthe intended capture and output equipment, as well as appropriate forthe intended use of the image.

Exact and Linear Reproduction

Exact and linear reproduction are where the reproduction and originalare identical according to some objective, physical measurementcriteria. If absolute measurement values are identical, the reproductionis exact; if relative measurement values are identical, the reproductionis linear. Exact reproduction is rarely practical, or even appropriate,with reproductions of scenes because of the differences between thescene illumination conditions and the reproduction viewing conditions.Reproductions of hardcopy on media similar to the original may be exactreproductions, because the original and reproduction can be viewed underthe same conditions. However, in such cases exact and linearreproduction are functionally identical, since the measures on whichlinear reproduction is typically based are relative to the media white.

Appearance Reproduction

Appearance reproduction is where the reproduction and original have thesame appearance when each is viewed under specific conditions. A linearmatch is an appearance match if the original and reproduction are onidentical media and are viewed under the same conditions, and a linearcalorimetric match is an appearance match if the white reference andviewing conditions remain constant. Currently, the only way to producean appearance match under any condition is with manual, trial-and-errorprocessing. Several appearance models have been developed that allowappearance matches to be produced under conditions that vary in specificways, but the accuracy of the matches varies to some extent with thedifferent models. Appearance models tend to be most successful indealing with changes in illumination chromaticity. Unfortunately, manyother changes are also important. In fact, one criteria for choosingphotographic dyes is to minimize changes in appearance due to changes inillumination chromaticity, as long as the observer is adapted to theillumination. Table 1 lists a number of factors affecting the appearanceof photographs.

TABLE 1 Factors Affecting Appearance Human Visual System Factors (forviewing both the scene and the reproduction) Flare in the Eye AdaptationState:  To the Overall Illumination Level  To the Illumination SpectralCharacteristics  Spatial Variations in Adaptation  IntermediateAdaptation to Multiple Conditions Factors Relating to Characteristics ofthe Scene or Original (as viewed by the observer, as opposed to a cameraor scanner) Overall Illumination Level Illumination SpectralCharacteristics Colorants Used (if known) Dynamic Range Scene Key (high-or low-) Scene Content Factors Relating to Characteristics of theReproduction Overall Illumination Level Illumination SpectralCharacteristics Dynamic Range and Surround Media Type - Surface orIlluminant Mode Surface Reflections and Veiling Glare Media ColorSynthesis Characteristics - Base Material and Colorant Gamut

Preferred (Pictorial) Reproduction

In photography, the most common reproduction goal is preferredreproduction, where the aim is to produce the most pleasing imageregardless of how well it matches the original. Since there is not anappearance model that deals extensively with dynamic range changes, andsince preferred reproduction is highly dependent on dynamic range, it isdifficult to say how much of preferred reproduction is an attempt toproduce a dynamic range appearance match. If a truly comprehensiveappearance model is developed, preferred reproduction may reduce toappearance matching with a slight s-shaped tone reproduction overlay andthe corresponding saturation boost. For the time being, preferredreproduction processing frequently offers the best path to excellentphotographic quality.

A minor consideration with preferred reproduction is that the nature ofthe reproduction desired depends slightly on the general media class. Innormal viewing contexts, illuminant mode color images, such as aredisplayed on a monitor, tend to require somewhat less s-shape thansurface mode color images, such as prints, with equivalent dynamicranges. However, quantification of preferred reproduction for digitalsystems is just beginning, and small effects are lost in the overalluncertainty.

Preferred Reproduction and Appearance Matching

Table 2 lists the most common reproduction goals for variousapplications of digital photography.

TABLE 2 Default Reproduction Goals for Digital Photography Input Form ->Output Form Scene Transparency Negative Print Transparency PreferredLinear Preferred Appearance Print Preferred Appearance Preferred Linear

In table 2, the default reproduction goal for producing a transparencyfrom a print, or vice versa, is to produce an appearance match. Strictlyspeaking, the means for achieving this has not been developed becausethese media have significantly different dynamic ranges when viewedunder typical conditions. However, a roundabout approach can be used toachieve the desired result. If it is assumed that the original exhibitspreferred reproduction, it is possible to undo this reproduction back toa linear space, and then implement preferred reproduction on the newmedia. The result will be very close to an appearance match. This typeof processing can be done using one LUT in an appropriate, standard RGBcolor space, assuming the output device can render the RGB datacorrectly.

sRGB Color Space Processing

Device performance optimization requires that pictorial processingalgorithms can interpret the meaning of the digital image data they arepresented. If the processing algorithms are specific to particulardevices, there are various ways in which this information can becommunicated. Device-independent performance optimization requires thatthe meaning of the data be understandable regardless of the device. Theonly way to accomplish this is to establish some sort of standard dataspace.

Most current color management paradigms make use of a perceptualdevice-independent color space, such as CIE XYZ or CIE L*a*b*. Themotivation for this color space type is that images are meant to beviewed, and color descriptions based on psychophysical measurements bestpredict appearance. This approach is theoretically indisputable if it ispossible to construct transforms that accurately convert image data to apsychophysical space, and if the psychophysical space accuratelydescribes appearance. Unfortunately, it is not always possible toconstruct such transforms, and the lack of a totally comprehensiveappearance model may prevent current psychophysical descriptions frompredicting appearance. The use of strict perceptual color spaces canalso result in fairly intensive processing requiring high precision,since the image data may be transformed through a non-native state.

Alternatives to perceptual color spaces are physically standardized, butmore device native “color” spaces. Such spaces describe the physicalmeaning of the data. It may also be possible to correlate these physicaldescriptions to current appearance model descriptions for limited setsof viewing conditions. Obvious candidate spaces of this type arestandard RGB spaces. Of those available, the most appropriate are themonitor spaces. Monitor spaces have wide gamuts, most images are viewedon a monitor at some point, a great deal of manual processing isaccomplished using monitor feedback, and an internationally standardizedmonitor space already exists, as well as correlations to appearanceunder specific viewing conditions. Standard monitor data, when printedon photographic media using devices with independent channels, alsotends to correlate reasonably well with Status A densitometry. Thismeans that photographically derived preferred reproduction models can beapplied. It is also interesting to note that recent work in the colorappearance area is indicating that the use of spectrally sharpenedvisual response functions is advantageous. These functions are muchcloser to RGB than the unsharpened visual (cone) response functions.Table 3 summarizes the advantages and disadvantages of the two types ofstandard color data spaces.

TABLE 3 Advantages and Disadvantages of Perceptual and sRGB Color SpacesCIE XYZ and L*a*b* Color Spaces Advantages Excellent color appearancereproduction if the capture is colorimetric or the colorants used in theoriginal are known, and the viewing conditions and media dynamic rangeare appropriate. Can reproduce color using unusual colorants as long asthe viewing conditions and media dynamic range are appropriate. L*a*b*is reasonably uniform perceptually. Disadvantages The color reproductionaccuracy advantage is lost if the capture is not colorimetric or thecolorants used in the original are not known, as is usually the casewith digital cameras. Color appearance prediction may be poor if theoutput media dynamic range and/or viewing conditions are significantlydifferent from the original. Processing may be more extensive andrequire higher precision. No model is available for preferredreproduction. If all the gamut benefits are to be realized, the imagedata may need to be stored at high precision, or the raw data storedwith a transform. sRGB Color Space Advantages Similar to many devicenative color spaces. Transformations to sRGB tend to be simpler, moreaccurate, and require less precision for storage. It is less necessaryto save the raw data with a transform. Transformations from sRGB tooutput device spaces also tend to be simpler. Since sRGB image data canalso be described perceptually, the advantages of the perceptual colorspaces can be applied. Photographic preferred reproduction models can beapplied. Reasonably uniform perceptually. Relatively independentchannels help with signal-to-noise issues in capture. May be similar tothe spectrally sharpened tristimulus metrics to be used in futureappearance models. Disadvantages Colors that are out of the monitorgamut are expressed using negative values, requiring larger datavolumes.

The standard monitor data approach provides a common ground to linkperception to the native physical behavior of devices based on RGB andCMY capture or colorants. Most devices that produce outputs of varyingdynamic range use colorants of these types. Output devices that useother colorants, but have dynamic ranges of around 100:1 can also beaccommodated since the monitor data can be correlated with perceptualmetrics at the fixed dynamic range. Various forms of standard monitorRGB have been successfully used by practitioners of digital photographyfor several years. Recently, a few major corporations have formalizedthis approach by proposing a specific standard monitor color space,sRGB. This proposal paves the way for the use of a standard RGB spacefor pictorial digital image processing (PDIP).

Digital Camera Processing Pipeline

The following is one embodiment of a proposed optimized pipeline forPDIP. It combines many aspects of my invention that are not required tobe used together. Specific capture and output devices, quantities andcolor spaces included in this pipeline are exemplary in nature and arenot intended to limit the scope of the invention in any way.

Preliminary Measurements

The gathering of preliminary information is shown schematically in FIGS.1 and 2 and is represented by boxes 100 and 101-109.

1. Set the camera gain(s) and offset(s) as they will be set during use,hopefully the optimum settings. Ideally, the offset should be set sothat a focal plane exposure of zero produces a digital level of zeroafter bias and dark current subtraction. The spacing of the levelsshould be chosen so that virtually all the information the camera cancapture is recorded with minimal quantization error. A rule of thumb isthat the standard deviation, expressed in digital levels, of any evenfocal plane exposure, should be at least 0.5 after all fixed patternnoise removal processing. Higher standard deviations are alsoacceptable, but will require more bits for image data storage.

2. Determine the camera fixed pattern noise characteristics, such asdark current and pixel sensitivity non-uniformity.

3. Measure the camera response characteristics to determine the focalplane OECF for each channel for each illumination type of interest.Variations due to illumination type can usually be dealt with using asingle OECF curve shape and channel multipliers. For example, aparticular camera may have relative channel sensitivities of 0.6, 0.9,and 1 with 5500K daylight illumination and 0.2, 0.7, and 1 with 3200Ktungsten illumination. If channel multipliers are used, it is possibleto determine a mathematical model to predict multipliers forintermediate illumination types.

4. Determine the camera OECFs (101) for a variety of scenes with knownspatial and spectral (or channel) radiance distributions. Use thisinformation in conjunction with the focal plane OECF measurements todevise a model that estimates flare and other image-dependentnon-linearities based on focal plane image statistics (102).

5. Determine a matrix to transform the linearized camera data into thestandard linear RGB space, using the methods described in the (mostcurrent version of the draft or international standard derived from the)new ISO work item proposal draft.

6. Measure the linearized camera spatial frequency responses and noisepower spectra for each channel in the horizontal and verticaldirections. Determine a reasonable processing kernel size and constructa maximum information throughput spatial reconstruction kernel. Notethat each channel may require a different reconstruction kernel, andthat the ideal reconstruction kernel may vary with focal plane exposure.

7. Measure the neutral EOCF, the spatial frequency response, and thenoise power spectrum of the output devices on which the image data maybe rendered. If an output device has minimal noise, it may not benecessary to measure the noise power spectrum. Spatial frequencyresponse and noise power spectrum measurements are also not necessarywith half-toning output devices that incorporate perceptual noisesuppression and sharpening into the half-toning algorithm. If the outputdevice is not known, the EOCF for sRGB can be used.

Processing Step 1: Determination of Flare and Scene Key

1. Divide up the image data into the color channels, if required (202).

2. Pixel average (boxcar filter (301, 301′) and sub-sample (302, 302′))each channel to obtain scaled versions of the original image, andpreferably store the scaled images. Other blur filters can be used, suchas Gaussian and median. The scaled images are preferably reduced imagesof 10,000 to 20,000 digital values. Previous work has indicated thatreduced images with a short dimension of about 100 pixels are optimalfor most subject matter and output forms. The pixel averaging is done inwhat is assumed to be an approximately logarithmic gamma type cameradata space because geometric means are preferable to arithmetic means.If the camera is linear, it would be best to convert the digital valuesto a log or gamma space prior to the pixel averaging, and then backagain, although the effect of this rather intensive additionalprocessing should be minimal.

3. Transform the reduced image values to focal plane exposures using theinverse focal plane OECFs for each channel, using the appropriate gainsand/or channel multipliers for the scene or original illumination type(401).

4. Determine the minimum, maximum, and mean exposures, and the minimum,maximum, and mean log exposures for each channel. Estimate theimage-specific camera OECFs for each channel based on these statistics(or a subset thereof) using the previously determinedflare/non-linearity model, and the focal plane OECFs for theillumination type used (402).

5. The scene key is determined by subtracting the average of the minimumand maximum log exposures, which is the expected mean log exposure, fromthe mean log exposure.

Processing Step 2: Determination of Scene Zone 1, Zone 5, and Zone 9Luminances

1. Transform the reduced image values into linear standard RGB scenevalues using the processing sequence as outlined below in ProcessingSteps 4 and 6 (404-407).

2. Combine the linear standard RGB scene values into a linear sceneluminance channel using an equation appropriate for the RGB color spaceused (408). For example, if the standard color space is sRGB, theluminance conversion equation from ITU-R BT.709 is appropriate.

3. The minimum reduced image luminance is assumed to be the scene Zone 1luminance, the arithmetic mean luminance the Zone 5 luminance, and themaximum luminance the Zone 9 luminance (409).

TABLE 4 Zone Designations of Perceptual Tone Categories Zone 0 absoluteblack or maximum density. Zone 1 edge of detail in black. Zone 2 texturein black. Zone 3 average dark objects. Zone 4 dark midtones. Zone 5medium midtone or middle gray. Zone 6 light midtones. Zone 7 averagelight objects. Zone 8 texture in white. Zone 9 edge of detail in white.Zone 10 absolute white or base white (with any fog or stain).

Processing Step 3: Determination of Output Table

1. Select the output device and pixel pitch for the desired rendering.If the output device is not known, assume a standard monitor asrepresented by sRGB.

At this point, the method proceeds along either a linear reproductionpath or a preferred reproduction path, with the desired path dependingon the particular application.

Linear Reproduction:

2a. Determine the digital level that produces a nominal 20% reflectance,or 20% transmittance relative to the base transmittance, or 20% of thewhite luminance, on the output device. This is designated as the midtonereflectance level. On devices with dynamic ranges different from the100:1 total dynamic range specified for sRGB, or devices which have thesame dynamic range but different viewing conditions, the perceptualmidtone reflectance, transmittance, or luminance factor may be differentfrom 20%.

3a. Determine a LUT that will produce an output with reflectances (ortransmittances or luminance factors, as appropriate) that are a constantmultiplier of the scene luminances, with the constant chosen so that theZone 5 scene luminance reproduces at the medium midtone. The input tothis LUT will be standard linear RGB scene values, the output will bestandard (but not necessarily linear, depending on the output deviceEOCF) RGB digital code values.

Preferred Reproduction:

It should be understood that the following description of preferredreproduction is only one way to accomplish preferred reproduction in myoverall method. Many other preferred reproduction methods can be used inthe overall strategy and still fall within the scope of my invention. Aswill be seen and as indicated above, the preferred reproduction modelcan even be a linear reproduction model or an appearance reproductionmodel. It should also be understood that where I discuss reproductionand other curves, I use “curve” in a broad sense since these are verycomplex, multidimensional mathematical functions. My use of the word“curve” includes look-up tables (LUTs), transformations, and matrices.

2b. Determine the maximum Zone 1 density capability of the outputmedium. This will be 90% of the maximum density capability for mostmedia. For self-luminous displays, such as monitors, “density” is equalto the negative base ten logarithm of the luminance factor of themonitor, relative to the monitor white point.

3b. Determine the viewing condition illumination level and ambientillumination level. This step may not always apply, depending on theoutput medium to be used, as well as other factors.

4b. Determine the lesser of: the maximum Zone 1 density capability ofthe output medium, or the desired Zone 1 density based on the viewingconditions. This will be the desired Zone 1 density.

5b. Determine the desired Zone 9 density, which is typically 0.04 abovethe minimum density for reflection hardcopy and monitors, and 0.06 abovethe minimum density for transparencies to be viewed on a light table orprojected in a darkened room.

6b. Calculate the preferred reproduction relationship between the sceneluminances and output densities. The details of this calculation are asfollows:

6.1 b. The following quantities are needed for the calculation of thescene specific preferred tone reproduction curve:

Zone 1 Log Luminance—Z1logL

Zone 9 Log Luminance—Z9logL

Mean Log Exposure—{overscore (logH)}

Expected Mean Log Exposure—({overscore (logH)})

Zone 1 Output Density—Z1D

Zone 9 Output Density—Z9D

6.2b. The quantities listed in the preceding step can be used tocalculate the following important values:

Scene Pictorial Dynamic Log Range—ΔlogL

ΔlogL=Z 9logL−Z 1logL  (1)

Output Pictorial Dynamic Log Range—ΔD

ΔD =Z 1 D−Z 9 D  (2)

Flex Factor Multiplier—FFM

$\begin{matrix}{{FFM} = \frac{{\Delta \quad \log \quad L} - {\Delta \quad D} + 2}{2.34}} & (3)\end{matrix}$

Shift Factor Multiplier—SFM

$\begin{matrix}{{SFM} = \frac{\overset{\_}{\log \quad H} - {\langle\overset{\_}{\log \quad H}\rangle}}{0.6}} & (4)\end{matrix}$

The preferred tone reproduction curve will be determined by adding anS-shaped flex to a reproduction curve that is linear with respect toscene log luminance and output density. The amount of flex depends onthe scene and output dynamic ranges. The flexed curve will then beshifted to compensate for low or high mean reflectances (as determinedusing the reduced image log exposure statistics).

6.3b. The manipulation of the base reproduction curves is accomplishedusing zones. Numerical values are added to the base zone values toproduce the desired zone values. The normalized base zone values,without any flex or shift, are provided in tables 5 and 6.

TABLE 5 Base Zone Log Luminances (BZLLs) Zone 1 Zone 2 Zone 3 Zone 4Zone 4.5 Zone 5 Zone 6 Zone 7 Zone 8 Zone 9 0 0.1613 0.3118 0.48390.5645 0.6452 0.7688 0.8763 0.957 1

TABLE 6 Base Zone Densities (BZDs) Zone 1 Zone 2 Zone 3 Zone 4 Zone 4.5Zone 5 Zone 6 Zone 7 Zone 8 Zone 9 1 0.8387 0.6882 0.5161 0.4355 0.35480.2312 0.1237 0.043 0

6.4b. The amount of flex in the desired tone reproduction curve is basedon the ratio of the scene pictorial dynamic range to the outputpictorial dynamic range. The flex is applied by adding flex zone valuesto the normalized base zone log luminances (BZLLs) listed in table 5.

6.4.1b. The standard flex factors (SFFs), listed in table 7, are for astandard scene pictorial dynamic range of 160:1 and a standard outputpictorial dynamic range of 72:1.

TABLE 7 Standard Flex Factors (SFFs) Zone 1 Zone 2 Zone 3 Zone 4 Zone4.5 Zone 5 Zone 6 Zone 7 Zone 8 Zone 9 0 0.087 0.086 0.036 0.004 −0.022−0.051 −0.063 −0.048 0

6.4.2b. The flex zone values are determined by multiplying the SFFs bythe FFM.

6.4.3b. The scene specific preferred zone log luminances are determinedby adding the flex zone values to the BZLLs listed in table 5,multiplying the sums by the AlogL, and adding the Z1logL (see equation5).

ZoneLogLuminances=Z 1logL+ΔlogL(BZLL's+FFM(SFF's))  (5)

6.5b. The amount of shift in the desired tone reproduction curve isbased on the difference between the mean log exposure and the expectedmean log exposure.

6.5.1b. If the mean log exposure is lower than the expected mean logexposure, the scene is a low key scene. The standard low key shiftfactors (SFs) are listed in table 8.

TABLE 8 Standard Low Key Shift Factors Zone 1 Zone 2 Zone 3 Zone 4 Zone4.5 Zone 5 Zone 6 Zone 7 Zone 8 Zone 9 0 0.1075 0.1344 0.1129 0.09140.0753 0.0538 0.0323 0.0108 0

These factors are based on a mean log exposure that is 0.6 log unitslower than the expected mean log exposure.

6.5.2b. If the mean log exposure is higher than the expected mean logexposure, the scene is a high key scene. The standard high key SFs arelisted in table 9.

TABLE 9 Standard High Key Shift Factors Zone 1 Zone 2 Zone 3 Zone 4 Zone4.5 Zone 5 Zone 6 Zone 7 Zone 8 Zone 9 0 0.0323 0.0484 0.0699 0.08060.086 0.086 0.0753 0.0484 0

These factors are based on a mean log exposure that is 0.6 log unitshigher than the expected mean log exposure.

6.5.3b. The shift zone values are determined by multiplying theappropriate standard shift factors by the SFM. The sign of the SFM isalso used to determine which set of shift factors to use. A negativesign means that the low key shift factors should be used, and a positivesign means that the high key shift factors should be used.

6.5.4b. The scene specific preferred zone densities are determined byadding the shift zone values to the base zone densities (BZDs) listed intable 6, multiplying the sums by the AD, and adding the Z9D (seeequation 6).

ZoneDensities =Z 9 D+ΔD(BZD's +|SFM|AppropriateSF's)  (6)

7b. Determine a LUT that will produce preferred reproduction on theselected output device, or a standard monitor.

Processing Step 4: Scene Linearization

1. Subtract and divide out the fixed pattern noise (if not alreadydone).

2. Construct input linearization tables by taking each possible digitalvalue through the image-specific inverse camera OECFs.

3. Convert the pixel digital values to linear scene channel radiances.

Processing Step 5: Spatial Restoration

Note: This step is placed here because most spatial restorationtechniques assume the data is in a linear radiance space. Also, it isadvantageous to perform the spatial reconstruction and thetransformation to the standard linear RGB color space in the sameoperation. This processing should be placed elsewhere in the pipeline ifthe reconstruction algorithm is designed to deal with nonlinear data,either as output by the capture device, or as will be input to theoutput device.

1. Apply the maximum information throughput (or other) spatialreconstruction kernel. As noted above, this kernel may also perform thetransformation to a standard linear RGB space (see Processing Step 6).

2. Apply any morphological or other nonlinear processing to the image toreduce artifacts (most common with CFA camera data).

Processing Step 6: Transformation to a standard linear RGB color space

Note: As stated above, this step and step 5 may be done simultaneously.

1. Apply the linear radiance to standard linear RGB color spacetransformation matrix.

Processing Step 7: Output Processing

1. Apply the desired output LUT. The data output by this step will bestandard (but probably not linear) RGB digital code values. If theoutput EOCF is the sRGB EOCF, the data will be sRGB data.

2. Apply any subsequent output processing, such as sharpening, noisereduction, transformation of the standard RGB data to another colorspace, application of output device-specific color LUTs, half-toning,etc.

Subsequent Processing

Image data that has been processed to a particular reproduction goal onone output device can be processed for another output device by undoingthe processing back to the point where the processed data is common toboth output devices. Changes in reproduction goal are similar. A newreproduction goal on the same output device can be viewed as a differentoutput device by the processing.

Processing Strategy for Scanning

Scanners can be viewed as digital cameras that are used to makereproductions of transparencies and prints, or as part of digital imagecapture systems that use film for the initial capture. If thereproduction goal is relative to the transparency or print, the formerviewpoint is correct, and the scanner data can be processed in the sameway as digital camera data. If the reproduction goal is relative to theoriginal scene, the latter viewpoint is more appropriate. In this caseit is necessary to consider the film camera, film, film processing, andscanner as part of the capture system. The linearization and colorcharacterization must be performed on the entire system.

Image data from scans of negatives must generally be related to thescene, so negative film capture should be considered as part of a largersystem. However, the linearization and color characterization offilm/scanner capture systems is complicated by film/process systemvariability. Also, it is important to remember that film does notcapture calorimetric information about the scene. Even if the colorantsused in the scene are known and limited in number to the number of filmcapture channels, film/process system variability typically preventsaccurate determination of calorimetric values. It is better to attemptto produce the desired reproduction directly from the film capturechannels, which are usually RGB.

With scans of negatives, it is also possible to relate a reproductiongoal to a conventional photographic print of the negative. This isequivalent to relating to the scene, and then using the negative-printsystem tone and color reproduction characteristics as the preferredreproduction model. However, such a model will be limited bycharacteristics of conventional photographic systems, some of which maynot be desirable and can be ignored in digital processing. Another usefor conventional photographic system tone and color reproduction modelsis to undo the preferred reproduction applied by conventionalphotographic systems so that the scanner data can be related to thescene. However, choosing to relate to the scene instead of thetransparency or print implies that the image was selected for the scenecontent, rather than for how the scene reproduced in the transparency orprint. The appropriate viewpoint must be chosen for the processingpipeline to create the desired reproduction.

In professional photography, most scenes are captured on transparencyfilm, and images are selected, at least to some extent, based on theappearance of the transparency, as opposed to the scene content. Thetransparency may therefore be the original to be reproduced. Thisreproduction is aided by the fact that, since the colorants used in thetransparency are known and limited, it is possible to obtaincalorimetrically accurate values from the scanner data. If thetransparency is to be printed on reflective media, an appearance matchmust still be created using standard RGB channels, but the calorimetricaccuracy will result in a better appearance match (to the transparency,not the scene) than if the RGB channels were not related to colormatching functions. However, if the transparency is reproduced onanother type of media, either reflective or transmissive, matching ofthe colors may not be preferred since the colors in the originaltransparency may be preferred only on the particular transparencymaterial.

In amateur photography, most scenes are captured on negative film, andimages are selected, at least initially, based on the appearance of thescene. In some respects negatives are better suited for film capturethan positives. They have a larger input dynamic range, more exposurelatitude, and the unwanted absorptions of the film dyes are correctedusing masks, making the RGB channels in a negative more orthogonal.Also, the output dynamic range of negatives is lower than withtransparencies, reducing flare and the dynamic range requirements forthe scanner. Unfortunately, many scanners have difficulty producing goodimage data from negatives because they are designed to scantransparencies. The level spacing is too large for the lower dynamicrange, and the processing software does not know what to do with thereversed and offset RGB color data. Another complication is thatnegative dyes are designed to modulate the red channel information atlonger wavelengths. Negatives should be scanned using channels withspectral sensitivities that allow Status M densities to be obtained, asopposed to the Status A densities appropriate for transparencies.

Print scanning is complicated by surface reflections which are highlydependent on the scanner optical geometry. For preferred reproduction,it is best to scan the print in a way that minimizes surfacereflections, such as by using the geometry specified for reflectiondensitometry measurements. If linear colorimetric reproduction isdesired, it is necessary to simulate the viewing geometry in thescanner. Ideally, different scanner geometries should be used fordifferent viewing conditions. This is rarely practical, however, and themost common measurement approach for calorimetric scannercharacterization is to use an integrating sphere, frequently with agloss trap. However, measurements taken using this type of geometry willbe inferior for producing preferred pictorial reproduction. Anotherapproach is to use the geometry for reflection densitometry and simulateveiling glare. This approach is rare because a model that can simulatethe veiling glare characteristics of the material scanned is required.

User Adjustments

A comprehensive strategy for PDIP must allow for user input to the finalresult. No automated routine can account for individual taste. Also, thestrategy presented here does not deal with the nature of objects inimages. These objects (such as people) can have a significant effect onthe desired reproduction. Object recognition and classification isprobably the next frontier in automated image processing.

Current processing software does not provide for the quick and easyadjustment of reproduction. This is because it is necessary to have aprocessing approach in place before user adjustments based on physicalmeasurements and models can be added. Automated image processing topreferred reproduction is a major advance in digital photography, butquick, easy, and intuitive tweaking of the result is almost equallyimportant. Table 10 lists a variety of user adjustment options. Theseoptions are divided into levels, so that the novice user will see onlythe simple controls, while more advanced users can choose to view moreoptions. Structures of this type are common in many softwareapplications. In all cases it is assumed that visual feedback for allchoices is provided, that user preferences can be stored, that alltechnical and device information that can be transferred automaticallyis, and that the user can request that default values be selected wherenot provided and used for all values, for some values, or the interfaceask each time. Complete descriptions of how each adjustment affects theprocessing of the image data should be available for advanced users.

TABLE 10 Manual Adjustment Options Level O Choices: Illuminationsource - accept default or device estimation, specify type, or use aneutral balance feature. Reproduction goal - linear or preferred. Outputdevice (default to sRGB, or to a device specified in image file). Level1 Adjustments: Brightness Slider Linear reproduction - adjusts theoutput density of the scene arithmetic mean (Zone 5) luminance.Preferred reproduction (fine) - adjusts the amount of high- or low-keyshift. Preferred reproduction (coarse) - implements a gamma typebrightness shift. Contrast Slider Linear matching - disabled. Preferredreproduction (fine) - adjusts the amount of flex. Preferred reproduction(coarse) - adjusts the scaled image scale factor. Color Balance Slider(3 options) Adjust using color temperature slider. Adjust usingtri-linear coordinate (RGB) joystick. Adjust using rectangularcoordinate (opponent color) joystick. Sharpness Slider Level 2Adjustments: Linear reproduction Choose to base the midtone on thegeometric mean. Preferred reproduction For each channel, allowadjustment of the scaled image scale factor, Zone 1 and Zone 9 outputdensity, and flare factor. Choose to base the key on the arithmeticmean. Level 3 Adjustments: Apply user specified look-up-tables andmatrices Specify look-up-table to convert each channel to sceneradiance. Specify matrix to convert to different channels for output.Specify look-up-table to produce desired reproduction on output.

The tools and techniques required for the improved, efficient, andautomated processing of pictorial images are becoming available. Ifprocessing of this type is implemented, the quality of digitalphotographs should surpass that of conventional photographs in mostareas, resulting in rapid acceleration in the acceptance of digitalphotography.

LIST OF REFERENCE NUMERALS AND CORRESPONDING ELEMENTS OF EXEMPLARYEMBODIMENTS

100 Step of gathering preliminary information

101 Sub-step of determining oecfs and oecf inverses for capture device

102 Sub-step of constructing model

103 Sub-step of determining channel multipliers

104 Sub-step of determining transformation to intermediate color space

105 Sub-step of determining spatial reconstruction kernels

106 Sub-step of determining expected tone/color reproductioncharacteristics of output device

107 Sub-step of determining expected viewing conditions

108 Sub-step of determining output device/medium visual densitycapabilities

109 Sub-step of selecting preferred reproduction model

200 Step of providing original image

201 Sub-step of capturing with capture device or reading from file

202 Sub-step of dividing original image into channels, if required

203 Sub-step of removing noise, if required

300 Step of constructing scaled image/version of original image

301 Sub-step of spatially blurring original image

302 Sub-step of sub-sampling original image

400 Step of analyzing scaled image/version

401 Sub-step of transforming scaled image into focal plane data

402 Sub-step of determining significant statistical values of focalplane data

403 Sub-step of determining flare characteristics or non-linearcharacteristics of original image

404 Sub-step of determining input linearization information (luts orfunctions)

405 Sub-step of applying input linearization information to scaledversion to produce linear scene integrated channel radiance data

406 Sub-step of normalizing scaled version channel radiance data

407 Sub-step of transforming normalized/balanced channel radiance datainto intermediate color space to produce transformed normalized scaledversion data

408 Sub-step of combining all channels of transformed normalized scaledversion data into a scene luminance channel

409 Sub-step of determining significant statistical values of sceneluminance

410 Sub-step of selecting desired manner of tone reproduction

411 Sub-step of calculating desired tone reproduction

412 Sub-step of determining eocf and inverse eocf of output device

413 Sub-step of determining output luts

500 Step of processing original image

501 Sub-step of applying input linearization information to originalimage to produced linearized channel data

502 Sub-step of applying spatial reconstruction kernels to linearizedchannel data to produce reconstructed channel data

503 Sub-step of multiplying reconstructed channel data be channelmultiplier to produce normalized/balanced channel data

504 Sub-step of transforming normalized/balance channel data tointermediate color space to produce intermediate channel data

505 Sub-step of applying output luts to intermediate channel data toproduce output/processed image

600 Step of constructing one or more models

601 Model is image-dependent flare model

602 Model is image-dependent non-linearity model

603 Model is image-dependent preferred reproduction model

604 Model is output-dependent preferred reproduction model

200′ Step of providing original image

300′ Step of constructing scaled image

301′ Sub-step of spatially blurring original image

302′ Sub-step of sub-sampling blurred image

400′ Step of deriving image statistics from scaled image

500′ Step of processing original image in accordance with statisticsderived from scaled image

501′ Sub-step of determining image-dependent linearization information

502′ Sub-step of determining image-dependent reproductioncharacteristics from image statistics

700 Step of determining scene luminance statistics of original

800 Step of determining output media density ranges

900 Step of determining preferred reproduction curves

901 Sub-step of construction preferred reproduction model

902 Sub-step of taking into account conditions under which image isviewed

1000 Step of applying preferred reproduction curves to original

1100 Step of determining whether original data is srgb

1200 Step of transforming original data to rgb (if necessary)

1300 Step of determining tone reproduction transformation

1310 Sub-step of determining relationship between luminance channels oforiginal and scaled images

1400 Step of applying tone reproduction transformation to each rgbchannel

1500 Step of specifying nature of original reproduction

1600 Step of obtaining image data related to original reproduction

1700 Step of transforming image data to appropriate color space

1800 Step of processing data for desired reproduction

1810 Desired reproduction uses same output media with differentreproduction

1820 Desired reproduction uses different output media with samereproduction

1830 Desired reproduction uses different output media with differentreproduction

1900 Step of providing original image

2000 Step of constructing reproduction model

2100 Step of processing original image in accordance with preferredreproduction model

2200 Step of allowing adjustment of processed image via adjustment ofparameter of preferred reproduction model

2210 Parameter is channel multiplier

2220 Parameter is scale factor

2230 Parameter is model value for scene key estimate of reproductionmodel

2240 Parameter is model value for scene luminance/output density rangerelation estimate

2250 Parameter is scene zone 1 output density

2260 Parameter is scene zone 5 output density

2270 Parameter is scene zone 9 output density

I claim:
 1. A method of pictorial digital image processing of anoriginal image comprising the steps of: collecting statistics of anoriginal image; obtaining density capabilities of an output device to beused for producing a reproduction; determining both an originalpictorial dynamic range from the statistics of the original image and areproduction pictorial dynamic range from the density capabilities ofthe output device; constructing a tone reproduction curve relating thestatistics of the original image to the visual density capabilities ofthe output device, based on a comparison between the original pictorialdynamic range and the reproduction pictorial dynamic range; andtransforming the original image into color space values, using the tonereproduction curve, for producing the reproduction with the outputdevice.
 2. The method of claim 1 in which the original image iscomprised of initial color space values of a scene captured by a capturedevice.
 3. The method of claim 1 in which the original image iscomprised of initial color space values of a focal plane image capturedby a capture device.
 4. The method of claim 1 in which the originalimage is comprised of capture device values.
 5. The method of claim 1 inwhich the original image is comprised of initial color space values ofan original captured by a capture device.
 6. The method of claim 5 inwhich the original is one of a print, a negative, and a transparency. 7.The method of claim 1 including a further step of obtaining informationabout characteristics of a capture device arranged for acquiring theoriginal image.
 8. The method of claim 7 in which the capture devicecharacteristics are obtained from information in an image file.
 9. Themethod of claim 1 in which the step of collecting statistics of theoriginal image includes first creating a scaled image from the originalimage and then collecting statistics of the scaled image.
 10. The methodof claim 9 in which the scaled image is a spatially blurred andsub-sampled image.
 11. The method of claim 1 in which the step ofcollecting statistics of the original image includes first applying atransform to the original image and then collecting statistics of thetransformed image.
 12. The method of claim 11 in which the transform isa conversion to a luminance image.
 13. The method of claim 11 in whichthe transform is based on physical characteristics of a capture devicearranged for acquiring the original image.
 14. The method of claim 13 inwhich the transform compensates for non-linearities of the capturedevice.
 15. The method of claim 13 in which the capture device valuesare transformed into an intermediate color space.
 16. The method ofclaim 5 in which information concerning processing used to create theoriginal image in initial color space values is ascertained by one ofthe steps of: (a) extracting information from an image file and (b)assuming a preferred reproduction based on a preferred reproductionmodel.
 17. The method of claim 16 in which the processing used to createthe original image is undone to produce estimated color space values ofa scene captured by the capture device.
 18. The method of claim 16 inwhich the processing used to create the original image is undone toproduce estimated color space values of a focal plane image captured bythe capture device.
 19. The method of claim 1 in which the step ofcollecting statistics of the original image includes estimating a key ofthe original image.
 20. The method of claim 9 in which the step ofdetermining the original pictorial dynamic range includes determiningthe original pictorial dynamic range from the statistics collected fromthe scaled version of the original image.
 21. The method of claim 20 inwhich the step of constructing the tone reproduction curve relates theoriginal pictorial dynamic range from the scaled version to thereproduction pictorial dynamic range.
 22. The method of claim 1 in whichthe density capabilities of the output device are assumed to be those ofa standard output device.
 23. The method of claim 1 in which the densitycapabilities of the output device are specified using some metric otherthan density.
 24. The method of claim 23 in which the densitycapabilities of the output device are specified using one of reflectancevalues, transmittance values, and luminance factors.
 25. The method ofclaim 1 in which the step of determining the original pictorial dynamicrange includes adjusting the dynamic range obtained from the statisticsof the original image to obtain a different original pictorial dynamicrange.
 26. The method of claim 25 in which the adjustment is determinedby a preferred reproduction model.
 27. The method of claim 25 in whichthe adjustment is user controlled.
 28. The method of claim 1 in whichthe step of determining the reproduction pictorial dynamic rangeincludes adjusting an output medium dynamic range obtained from thedensity capabilities to obtain a different reproduction pictorialdynamic range.
 29. The method of claim 28 in which the adjustment isdetermined by a preferred reproduction model.
 30. The method of claim 28in which the adjustment is user controlled.
 31. The method of claim 1 inwhich the step of constructing the tone reproduction curve includesapplying an S-shaped flex adjustment to the tone reproduction curvebased on a difference between the original pictorial dynamic range andthe reproduction pictorial dynamic range.
 32. The method of claim 31including a further step of providing for user control of the flexadjustment to regulate contrast of the reproduction.
 33. The method ofclaim 1 in which the step of constructing a tone reproduction curveincludes applying a shift adjustment to the tone reproduction curvebased on the statistics from the original image.
 34. The method of claim33 including a further step of providing for user control of the shiftadjustment to regulate brightness of the reproduction.
 35. The method ofclaim 1 in which the step of constructing the tone reproduction curve isin part based on an estimate of a key of the original image.
 36. Themethod of claim 1 in which the tone reproduction curve is incorporatedinto a lookup table.
 37. The method of claim 36 including a further stepof providing for user control by manipulating the lookup tableincorporating the tone reproduction curve.
 38. The method of claim 1 inwhich the tone reproduction curve is at least in part defined bymathematical relationships relating the statistics of the original imageand the density capabilities of the output device.
 39. The method ofclaim 38 in which the tone reproduction curve is further defined by alookup table.
 40. The method of claim 38 including a further step ofproviding for user control by altering parameters of the mathematicalrelationships describing the tone reproduction curve.
 41. The method ofclaim 1 including a further step of obtaining information about viewingconditions under which the reproduction is intended to be viewed. 42.The method of claim 41 in which the viewing condition information isused to adjust the reproduction pictorial dynamic range.
 43. The methodof claim 41 in which the viewing condition information is used to adjustthe tone reproduction curve.
 44. The method of claim 1 in which the stepof transforming the original image includes applying the tonereproduction curve to a luminance channel.
 45. The method of claim 1 inwhich the step of transforming the original image includes applying thetone reproduction curve to color channels.
 46. The method of claim 1including a further step of obtaining a transform for the output deviceand the step of transforming the original image data includes applyingboth the tone reproduction curve and the transform for the outputdevice.
 47. The method of claim 46 in which the transform for the outputdevice is a transform from the color space values produced by applyingthe tone reproduction curve to values appropriate for producing thereproduction on the output device.
 48. The method of claim 46 in whichthe transform for the output device includes at least one of: a lookuptable of one or more dimensions and mathematical relationshipsspecifying the transform.
 49. The method of claim 46 in which the tonereproduction curve and transform for the output device are combined intoa single transform and applied to produce the reproduction.